Multi-modal fusion method for user authentication and user authentication method

ABSTRACT

A user authentication method includes receiving a first input image including information on a first modality; receiving a second input image including information on a second modality; determining at least one first score by processing the first input image based on at least one first classifier, the at least one first classifier being based on the first modality; determining at least one second score by processing the second input image based on at least one second classifier, the at least one second classifier being based on the second modality; and authenticating a user based on the at least one first score, the at least one second score, a first fusion parameter of the at least one first classifier, and a second fusion parameter of the at least one second classifier.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims under 35 U.S.C. § 119 to Korean PatentApplication No. 10-2015-0109712, filed on Aug. 3, 2015, in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference in its entirety.

BACKGROUND

1. Field

At least one example embodiment relate to a multi-modal method for userauthentication and a user authentication method.

2. Description of the Related Art

Significance of security authentication is increasing with thedevelopment of various mobile devices or wearable devices for example,smart phones. Authentication technology by biometric recognitionauthenticates a user using a fingerprint, an iris, a voice, a face, andblood vessels, since such biometric features used for authenticationvary from person to person, do not cause any inconvenience of carryingwith, have a minimal or reduced risk of being stolen or imitated, and donot change during a lifetime.

For the authentication technology by biometric recognition, anon-contact type recognition method, for example, face recognition andiris recognition, may be used in addition to a contact type recognitionmethod, for example, fingerprint recognition. When the non-contact typerecognition method is used, a user may freely move a hand using a mobilephone. In an example, an infrared ray (IR) camera is used for the irisrecognition, and an eye area may be detected in an IR image. Inaddition, a camera for self photographing is used for the facerecognition in lieu of requiring an additional device.

SUMMARY

Some example embodiments relate to a user authentication method.

In some example embodiment, the method may include receiving a firstinput image including information on a first modality, receiving asecond input image including information on a second modality,determining at least one first score by processing the first input imagebased on at least one first classifier, the at least one firstclassifier being based on the first modality, determining at least onesecond score by processing the second input image based on at least onesecond classifier, the at least one second classifier being based on thesecond modality, and authenticating a user based on the at least onefirst score, the at least one second score, a first fusion parameter ofthe at least one first classifier, and a second fusion parameter of theat least one second classifier.

The determining of the at least one first score may include extracting apredetermined subarea from the first input image, and determining ascore of the subarea using a classifier corresponding to the extractedsubarea among the at least one first classifier.

The determining of the at least one second score may include dividingthe second input image into a plurality of blocks, determining matchingscores between the blocks and enrolled images, and selectingpredetermined best scores among the matching scores.

Each of the first modality and the second modality may include at leastone of a face of the user, a fingerprint of the user, an iris of theuser, and a vein of the user, and the first modality may differ from thesecond modality.

The first fusion parameter and the second fusion parameter may bedetermined in advance based on a boosting scheme. The boosting schememay select a classifier having an optimal performance among candidateclassifiers based on a boosting parameter training data, update theboosting parameter based on an output of the selected classifier, anddetermine at least one of the first fusion parameter and the secondfusion parameter by determining a fusion parameter of the selectedclassifier based on the updated boosting parameter.

The authenticating of the user may include performing weighted summingof the first score and the second score based on the first fusionparameter and the second fusion parameter, and authenticating the userby comparing a result of the weighted summing to a predeterminedthreshold.

The at least one first classifier may be determined to correspond to atleast one predetermined subarea in the first input image. The at leastone second classifier may be determined based on a result of arrangingmatching scores between enrolled images and blocks to divide the secondinput image.

Other example embodiments relate to a multi-modal fusion method.

In some example embodiments, the method may include determining fusionparameters of candidate classifiers by iteratively selecting any one ofthe candidate classifiers including at least one first classifierconfigured to authenticate a user based on a first modality and at leastone second classifier configured to authenticate the user based on asecond modality. The fusion parameters may include weights of thecandidate classifiers.

The determining of the fusion parameters of the candidate classifiersmay include selecting a classifier having an optimal performance amongthe candidate classifiers based on a boosting parameter of trainingdata, updating the boosting parameter based on an output of the selectedclassifier, and determining a fusion parameter of the selectedclassifier based on the updated boosting parameter. The boostingparameter may be determined in a previous iteration

The selecting of the classifier may include selecting a classifierhaving an optimal performance index determined based on the boostingparameter and a result of classifying the training data.

The updating of the boosting parameter may include decreasing a value ofthe boosting parameter of the training data of which a recognitionresult of the selected classifier is true, and increasing the value ofthe boosting parameter of the training data of which the recognitionresult of the selected classifier is false.

The multi-modal fusion method may further include at least one ofdetermining a final parameter set based on the fusion parametersdetermined to correspond to each of the candidate classifiers when thedetermining of the fusion parameters is iteratively performed by apredetermined number, and determining a final parameter set based on thefusion parameters determined to correspond to each of the candidateclassifiers when a recognition rate determined in the determining of thefusion parameters is greater than or equal to a predetermined threshold.

The at least one first classifier may be determined to correspond to atleast one predetermined subarea in an input image. The at least onesecond classifier may be determined based on a result of arrangingmatching scores between enrolled biometric information and blocks todivide input biometric information.

Additional aspects of example embodiments will be set forth in part inthe description which follows and, in part, will be apparent from thedescription, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of example embodiments ofthe inventive concepts will become more apparent by describing in detailexample embodiments of the inventive concepts with reference to theattached drawings. The accompanying drawings are intended to depictexample embodiments of the inventive concepts and should not beinterpreted to limit the intended scope of the claims. The accompanyingdrawings are not to be considered as drawn to scale unless explicitlynoted.

FIG. 1 illustrates an example of authenticating a user based on amulti-modal fusion method according to at least one example embodiment;

FIG. 2 illustrates an example of a configuration of a multi-modal fusionapparatus according to at least one example embodiment;

FIG. 3 is a flowchart illustrating a multi-modal fusion method accordingto at least one example embodiment;

FIG. 4 is a flowchart illustrating a method of determining fusionparameters of candidate classifiers according to at least one exampleembodiment;

FIGS. 5A and 5B are graphs illustrating examples of distributions ofoutput scores of classifiers based on a first modality and a secondmodality according to at least one example embodiment;

FIGS. 6A through 6C are graphs illustrating examples of a histogramdistribution of a homogeneous modality according to at least one exampleembodiment; and

FIG. 7 is a flowchart illustrating a user authentication method based ona multi-modal fusion method according to at least one exampleembodiment.

DETAILED DESCRIPTION

Detailed example embodiments of the inventive concepts are disclosedherein. However, specific structural and functional details disclosedherein are merely representative for purposes of describing exampleembodiments of the inventive concepts. Example embodiments of theinventive concepts may, however, be embodied in many alternate forms andshould not be construed as limited to only the embodiments set forthherein.

Accordingly, while example embodiments of the inventive concepts arecapable of various modifications and alternative forms, embodimentsthereof are shown by way of example in the drawings and will herein bedescribed in detail. It should be understood, however, that there is nointent to limit example embodiments of the inventive concepts to theparticular forms disclosed, but to the contrary, example embodiments ofthe inventive concepts are to cover all modifications, equivalents, andalternatives falling within the scope of example embodiments of theinventive concepts. Like numbers refer to like elements throughout thedescription of the figures.

It will be understood that, although the terms first, second, etc. maybe used herein, to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of example embodiments of theinventive concepts. As used herein, the term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it may be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between” versus “directly between”, “adjacent” versus “directlyadjacent”, etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the inventive concepts. As used herein, the sin forms“a”, “an” and the are intended to include the plural forms as well,unless the context clearly indicates otherwise. It will be furtherunderstood that the terms “comprises”, “comprising,”, “includes” and/or“including”, when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and groups thereof.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Example embodiments of the inventive concepts are described herein withreference to schematic illustrations of idealized embodiments (andintermediate structures) of the inventive concepts. As such, variationsfrom the shapes of the illustrations as a result, for example, ofmanufacturing techniques and/or tolerances, are to be expected. Thus,example embodiments of the inventive concepts should not be construed aslimited to the particular shapes of regions illustrated herein but areto include deviations in shapes that result, for example, frommanufacturing.

Although corresponding plan views and/or perspective views of somecross-sectional view(s) may not be shown, the cross-sectional view(s) ofdevice structures illustrated herein provide support for a plurality ofdevice structures that extend along two different directions as would beillustrated in a plan view, and/or in three different directions aswould be illustrated in a perspective view. The two different directionsmay or may not be orthogonal to each other. The three differentdirections may include a third direction that may be orthogonal to thetwo different directions. The plurality of device structures may beintegrated in a same electronic device. For example, when a devicestructure (e.g., a memory cell structure or a transistor structure) isillustrated in a cross-sectional view, an electronic device may includea plurality of the device structures (e.g., memory cell structures ortransistor structures), as would be illustrated by a plan view of theelectronic device. The plurality of device structures may be arranged inan array and in a two-dimensional pattern.

Examples may be implemented as various types of products, for example,personal computers, laptop computers, tablet computers, smart phones,televisions, smart home appliances, intelligent vehicles, kiosks, andwearable devices. For example, the examples may be applied topre-processing of an image to authenticate a user using a smart phone, amobile device, and a smart home system. In the same manner, the examplesmay be applied to an intelligent vehicle system, and a payment servicethrough user authentication. Hereinafter, reference will now be made indetail to examples with reference to the accompanying drawings, whereinlike reference numerals refer to like elements throughout.

A multi-modal fusion apparatus, hereinafter, referred to as a “fusionapparatus”, may determine fusion parameters corresponding to a pluralityof modalities to authenticate a user based on a plurality of modalities.The fusion apparatus may be provided in a software module, a hardwaremodule, or various combinations thereof. For example, the fusionapparatus may be implemented based on various methods such as a singleprocessor-based system, a special purpose hardware-based system, asystem in which a processor and a hardware accelerator (HWA) arecombined, and a cloud computing system. For example, according to atleast one example embodiment, the fusion apparatus may include or beimplemented by one or more circuits or circuitry (e.g., hardware)specifically structured to carry out some or all of the operationsdescribed herein as being performed by a fusion apparatus (or an elementthereof). According to at least one example embodiment, the fusionapparatus may include or be implemented by a memory and one or moreprocessors executing computer-readable code (e.g., software) that isstored in the memory and includes instructions corresponding to some orall of the operations described herein as being performed by a fusionapparatus (or an element thereof). According to at least one exampleembodiment, the fusion apparatus may be implemented by, for example, acombination of the above-referenced hardware and processors executingcomputer-readable code. The term ‘processor,’ as used in the presentdisclosure, may refer to, for example, a hardware-implemented dataprocessing device having circuitry that is physically structured toexecute desired operations including, for example, operationsrepresented as code and/or instructions included in a program. Examplesof the above-referenced hardware-implemented data processing deviceinclude, but are not limited to, a microprocessor, a central processingunit (CPU), a processor core, a multi-core processor; a multiprocessor,an application-specific integrated circuit (ASIC), and a fieldprogrammable gate array (FPGA). Processors executing program code areprogrammed processors, and thus, are special-purpose computers.

A modality refers to an aspect in which a unique feature of a user ispresent. For example, the modality includes a face of the user, afingerprint of the user, an iris of the user, and a vein of the user. Aswill be described later, the fusion apparatus may determine a fusionparameter for fusing different modalities for a user authentication.Hereinafter, a method of determining a fusion parameter to authenticatea user based on a combination of a face and a fingerprint of a user, anda method of authenticating a user based on a fusion parameter will bedescribed with reference to the accompanying drawings. For ease ofdescription, it is assumed that a face of a user is used as a firstmodality and a fingerprint of the user is used as a second modality.However, examples are extended to use a combination of variousmodalities. For example, i) a face of a user and an iris of the user maybe used, ii) a fingerprint of a user and a vein of the user may be used,or iii) at least three different modalities may be used. Further,according to at least some example embodiments, a fusion parameter is aparameter used in a fusion process/operation. In more detail, whenmulti-model fusion based on a face and a fingerprint is performed by wayof weight summation, the value of Weight is the “fusion parameter.”

FIG. 1 illustrates an example of authenticating a user based on amulti-modal fusion method according to at least one example embodiment.Referring to FIG. 1, a face image 110 and a fingerprint image 130 inputfor a user authentication are illustrated.

A fusion apparatus extracts a feature of at least one predetermined or,alternatively, desired subarea in the face image 110 from the input faceimage 110. The subarea may be a predetermined or, alternatively, desiredarea in a face of a user, for example, a face, a contour of the face, anose, and an eye. Subareas may overlap.

The fusion apparatus may calculate a score indicating a similaritybetween a subarea and enrolled images by comparing pre-enrolled imagesto subareas extracted from the face image 110.

The fusion apparatus may calculate scores by comparing subareasextracted from the enrolled images of a face to the subareas extractedfrom the face image 110. The enrolled images of the face may be enrolledin advance, and subareas of the enrolled images may be extracted basedon an identical method of extracting the subareas of the face image 110.For example, the fusion apparatus may use, as a score, a normalizedcorrelation value between the subareas of the enrolled images and thesubareas of the face image 110, or use, as a score, a similarity basedon a size of an overlap area between the subareas of the enrolled imagesand the subareas of the face image 110. In an example, three subareas111, 113, and 115 for the face recognition are used, a number ofsubareas to be extracted for the face recognition is not limitedthereto.

The fusion apparatus determines scores Sface1, Sface2, and Sface3 bycomparing the subareas of the enrolled images to the subareas 111, 113,and 115 extracted from the face image 110. For example, each of thescores Sface1, Sface2, and Sface3 may be determined by different faceclassifiers corresponding to the subareas 111, 113, and 115,respectively.

The fingerprint image 130 may be a portion image corresponding to aportion of an entire fingerprint of the user. The fusion apparatusextracts block images 131, 132, 133, 134, . . . , and 138 by dividingthe input fingerprint image 130 into a plurality of blocks. The blockimages 131, 132, 133, 134, . . . , and 138 may be overlapped, or not beoverlapped.

The fusion apparatus may calculate scores Sfinger 1, Sfinger 2, . . . ,and Stinger n by comparing the enrolled images of the fingerprint to theblock images 131, 132, 133, 134, . . . , and 138 extracted from thefingerprint image 130. Such as the fingerprint image 130, the enrolledimages of the fingerprint may be portion images corresponding to aportion of the entire fingerprint.

In an example, when a number of block images is B, for example, anatural number greater than 1, and a number of enrolled image is E, forexample, a natural number greater than 1, the fusion apparatus maycalculate a score by comparing each of the enrolled images to each ofthe block images. Concisely, the fusion apparatus may perform comparingand image registration B×E times.

For example, the fusion apparatus may calculate scores Sfinger 1,Sfinger 2, . . . , and Sfinger 9 based on block pattern imageregistration between the enrolled images of the fingerprint and theblock images 131, 132, 133, 134 . . . , and 138. For example, scores maybe determined by a fingerprint classifier to perform comparing and imageregistration of block images and enrolled images.

The fusion apparatus may select scores SfingerTop 1, SfingerTop 2, andSfingerTop 3 in a predetermined or, alternatively, desired number, forexample, three, and the scores SfingerTop 1, SfingerTop 2, andSfingerTop 3 have highest values among the scores Sfinger 1, Sfinger 2,. . . , and Sfinger 9 calculated based on the block pattern imageregistration.

In general, since fingerprint recognition has a relatively small overlaparea between a distribution of positive samples and a distribution ofnegative samples compared to the face recognition, a recognition resultmay be clearly classified to be true or false. In an example, such afeature of a modality is used for calculating a final score by selectinghighest scores with respect to the fingerprint recognition.

In such an example, fusion authentication or multi-authentication may beperformed by multi-modal fusion based on a feature of each modality. Thefeature of each modality will be described with reference to FIG. 5.Also, a fusion authentication result of a face and a fingerprint will bedescribed with reference to FIG. 6.

The fusion apparatus may calculate a final score Sfinal, for example, afinal image registration result, by performing weighted summing of thescores Sface1, Sface2, and Sface3 associated with the face recognitionand the scores SfingerTop 1, SfingerTop 2, and SfingerTop 3 selectedassociated with the fingerprint recognition. The scores Sface1, Sface2,and Sface3 associated with the face recognition and weights wface1,wface2, wface3, wfingerTop 1, wfingerTop 2, and wfingerTop 3corresponding to each of the scores SfingerTop 1, SfingerTop 2, andSfingerTop 3 selected associated with the fingerprint recognition may belearned in advance. A method of learning weights will be described belowin detail.

The fusion apparatus may accept or reject user authentication inresponse to a result of comparing the final score Sfinal, for example,the final registration result, to a predetermined or, alternatively,desired threshold. Security levels may differ based on a type of anapplication to perform the user authentication. The threshold may bedetermined in advance based on a security level required based on a typeof an application. The security level may be expressed by variousmetrics, for example, a false acceptance rate (FAR) and a falserejection rate (FRR).

FIG. 2 illustrates an example of a configuration of a multi-modal fusionapparatus according to at least one example embodiment. Referring toFIG. 2, a fusion apparatus may allow fusion parameters to be learned.Here, the fusion parameters may be used for fusing a plurality ofcandidate classifiers 200.

The plurality of candidate classifiers 200 includes at least one firstclassifier 211 and at least one second classifier 212. The at least onefirst classifier 211 authenticates a user based on a first modality. Forexample, the first modality may be a face of a user. As described above,the at least one first classifier 211 is determined to correspond to atleast one predetermined or, alternatively, desired subarea in an inputimage.

The at least one first classifier 211 includes, for example, aclassifier 1 (c1), a classifier 2 (c2), and a classifier 3 (c3) for facerecognition. The classifier 1 (c1) recognizes an entire face contourarea, the classifier 2 (c2) recognizes an area including a nose, eyes,and lips of the user, and the classifier 3 (c3) recognizes an areaincluding the nose of the user.

The classifier 1 (c1), the classifier 2 (c2), and the classifier 3 (c3)may receive subareas extracted from the input image based on the firstmodality. For example, the classifier 1 (c1) receives a subarea 221, theclassifier 2 (c2) receives a subarea 231, and the classifier 3 (c3)receives a subarea 241. In an example, the classifier 1 (c1), theclassifier 2 (c2), and the classifier 3 (c3) may receive the input imagebased on the first modality and directly extract the subareas from thereceived input image.

The classifier 1 (c1) includes a feature extractor 222 and a matcher223. The feature extractor 222 extracts a feature vector from thesubarea 221. For example, the feature extractor 222 may extract thefeature vector from the subarea 221 based on a convolutional neuralnetwork (CNN). The feature vector includes a plurality of features. Thematcher 223 may match the feature vector extracted by the featureextractor 222 to a feature vector of a pre-enrolled face enrolled image.For example, the matcher 223 may calculate a score S1 indicating asimilarity between a subarea of the enrolled image and a subarea of theinput image based on a Euclidean distance between the subarea of theenrolled image and the subarea of the input image. The classifier 2 (c2)and the classifier 3 (c3) may calculate a score S2 and a score S3 basedon a similar method used by the classifier 1 (c1).

The at least one second classifier 212 authenticates the user based on asecond modality. For example, the second modality may be a fingerprintof the user. The at least one second classifier 212 is determined basedon a result of arranging image matching scores, hereinafter, referred toas matching scores, between enrolled biometric information and blocks todivide input biometric information. For example, the at least one secondclassifier 212 refers to the classifiers selected in FIG. 1.

The at least one second classifier 212 includes, for example, aclassifier 4 (c4), a classifier 5 (c5), and a classifier 6 (c6). Theclassifier 4 (c4), the classifier 5 (c5), and the classifier 6 (c6) mayreceive divided blocks from the input image based on the secondmodality. For example, from a fingerprint image, the classifier 4 (c4)receives a divided block 251, the classifier 5 (c5) receives a dividedblock 261, and the classifier 6 (c6) receives a divided block 271. In anexample, the classifier 4 (c4), the classifier 5 (c5), and theclassifier 6 (c6) may receive the input image based on the secondmodality and directly divide the received input image.

The classifier 4 (c4) includes a feature extractor 252 and a matcher253. The feature extractor 252 extracts a feature vector from the block251. For example, the feature extractor 252 may extract the featurevector from the block 251 based on a fast Fourier transform (FFT). Thematcher 253 may match a feature vector extracted by the featureextractor 252 to a feature vector of the enrolled biometric information.For example, the matcher 253 may calculate a score S4 using blockmatching between the block 251 and the enrolled biometric information.The classifier 5 (c5) and the classifier 6 (c6) may calculate a score S2and a score S3 based on a similar method used by the classifier 4 (c4).

In another example, the classifier 4 (c4), the classifier 5 (c5), andthe classifier 6 (c6) may include a matching device, instead of thefeature extractors 252, 262, and 272. The matching device may performimage registration on blocks and the enrolled biometric information. Forexample, the matching device may determine a transitional and rotationalrelationship between the blocks and the enrolled biometric information.In this example, the matcher 253, 263, and 273 may calculate asimilarity of the blocks and the enrolled biometric information based ona result of the image registration.

The fusion apparatus may determine a fusion parameter for fusing thecandidate classifiers 200 by iteratively selecting any one of thecandidate classifiers 200. The fusion parameter may include weights ofthe candidate classifiers 200. An operation of iteratively selecting anyone of the candidate classifiers 200 will be described with reference toFIG. 4.

In an example, a heterogeneous biometric recognizer is separated into aplurality of candidate classifiers thereby maximizing fusionperformance. For example, a fingerprint recognizer and a face recognizermay be separated into a plurality candidate classifiers for each area orfeature.

In the candidate classifiers 200, the six scores s1, s2, s3, s4, s5, ands6 are obtained. When the six scores s1, s2, s3, s4, s5, and s6 areassumed to be results of an independent classifier and performgentle-boost, n weak classifiers may be obtained.

The n weak classifiers may be a combination of binary classifiers. Forexample, a final registration result using the n weak classifiers may beexpressed as shown in Equation 1.

$\begin{matrix}{{g(x)} = {\sum\limits_{i = 1}^{n}\;\left( {{a_{i}\left( {{H_{i}(x)} > {th}_{i}} \right)} + b_{i}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Equation 1, g(x) denotes a final registration result, H_(i)(x)denotes a score output by an i^(th) weak classifier, thi denotes athreshold of the ith weak classifier, ai denotes a fusion parameter forthe ith weak classifier, and b_(i) denotes a bias value for the i^(th)weak classifier.

The expression (H_(i)(x)>th_(i)) is a term differing based on whetherthe score H_(i)(x) is greater than the threshold th_(i). For example,the term (H_(i)(x)>th_(i)) is +1 when the score H_(i)(x) is greater thanthe threshold th_(i), and term (H_(i)(x)>th_(i)) is −1 when the scoreH_(i)(x) is less than or equal to the threshold th_(i). The fusionapparatus may perform binary decision based on whether the finalregistration result g(x) is + or −.

As shown in Equation 1, when the score H_(i)(x) is less than or equal tothe threshold th_(i), the term (H_(i)(x)>th_(i)) is −1, so that thefinal registration result g(x) may be calculated to be small whencompared to the score H_(i)(x) being greater than the threshold th_(i).In an example, a combination of a linear classifier obtained as shown inEquation 2 may be used.

$\begin{matrix}\begin{matrix}{{g(x)} = {\sum\limits_{i = 1}^{n}\;\left( {{a_{i}\left( {{H_{i}(x)} > {th}_{i}} \right)} + b_{i}} \right)}} \\{\approx {\sum\limits_{i = 1}^{n}\;\left( {{a_{i}\left( {{H_{i}(x)} - {th}_{i}} \right)} + b_{i}} \right)}} \\{= {{\sum\limits_{i = 1}^{n}\left( {a_{i}{H_{i}(x)}} \right)} + {\sum\limits_{i = 1}^{n}\;\left( {{{- a_{i}}{th}_{i}} + b_{i}} \right)}}} \\{= {{{\sum\limits_{i = 1}^{n}\;\left( {a_{i}{H_{i}(x)}} \right)} + c} \approx {\sum\limits_{j = 1}^{n}\;\left( {a_{i}s_{i}} \right)}}}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In Equation 2 expresses H_(i)(x)=s_(i) and (a_(i)H_(i)(x))+c≈a_(i)s_(i).

In another example, the final registration result g(x) may be expressedas shown in Equation 3.

$\begin{matrix}\begin{matrix}{{g(x)} = {\sum\limits_{i = 1}^{n}\;\left( {{a_{i}\left( {{H_{i}(x)} > {th}_{i}} \right)} + b_{i}} \right)}} \\{\approx {\sum\limits_{i = 1}^{n}\;\left( {{a_{i}\left( \frac{{H_{i}(x)} - {th}_{i}}{\sigma_{i}} \right)} + b_{i}} \right)}} \\{= {\sum\limits_{i = 1}^{n}\;\left( {{\frac{a_{i}}{\sigma_{i}}{H_{i}(x)}} + d} \right)}} \\{\approx {\sum\limits_{i = 1}^{n}\;\left( {\frac{a_{i}}{\sigma_{i}}s_{i}} \right)}}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

In Equation 3, thi denotes an average of scores output by the ith weakclassifier. The operand oi denotes a score normalization parameter, andthe score normalization parameter may be a standard deviation of thescores output by the ith weak classifier.

In an example, a user may be authenticated based on whether the finalregistration result g(x) is greater than a pre-selected threshold th.The threshold th may be pre-selected based on test data. For example,the threshold th may be pre-selected based on an FAR required by anapplication.

In an example, as described above, the fusion apparatus may divide arecognition area for each modality, select a score suitable for eachmodality, and perform multi-authentication by applying the selectedscore to a fusion parameter.

For example, a multi-modal fusion in a learning-based method, such as aneural network and a soft vector machine (SVM), requires a functionrelated to an additional learning model, thereby increasing acomplexity. In an example, performance of the fusion apparatus mayeasily enhance without an additional complex calculation process by ascore level fusion.

FIG. 3 is a flowchart illustrating a multi-modal fusion method accordingto at least one example embodiment.

Referring to FIG. 3, in operation 310, a fusion apparatus determinesfusion parameters of candidate classifiers by iteratively selecting anyone of the candidate classifiers including at least one first classifierand at least one second classifier. The fusion parameters includeweights of the candidate classifiers.

In operation 320, the fusion apparatus verifies that operation 310 ofdetermining the fusion parameters is iteratively performed apredetermined or, alternatively, desired number of times, for example,20 times. When operation 310 is verified not to be iteratively performedthe predetermined or, alternatively, desired number of times, the fusionapparatus may perform operation 310 again.

In operation 330, when operation 310 is verified to be iterativelyperformed the predetermined or, alternatively, desired number of times,the fusion apparatus determines a final parameter set based on thefusion parameters determined to correspond to each of the candidateclassifiers. For example, the fusion apparatus may calculate a sum ofscores determined with respect to an identical classifier and determinea final parameter set by normalizing scores of each classifier. Forexample, the final parameter set may be in a form of {a1, a2 . . . ,an}.

In an example, the fusion apparatus may determine the final parameterset based on whether a recognition rate determined in operation 310 isgreater than or equal to a predetermined or, alternatively, desiredthreshold, other than based on an iteration number of performingoperation 310 of determining the fusion parameters. When the recognitionrate, for example, a performance index, determined in operation 310 isgreater than or equal to the predetermined or, alternatively, desiredthreshold, the fusion apparatus may determine the final parameter setbased on the fusion parameters determined to correspond to each of thecandidate classifiers. A reason of verifying whether the determinedrecognition rate is greater than or equal to the predetermined or,alternatively, desired threshold is for determining the final parameterset and completing learning of the fusion parameters for fusing thecandidate classifiers.

FIG. 4 is a flowchart illustrating a method of determining fusionparameters of candidate classifiers according to at least one exampleembodiment.

Referring to FIG. 4, in operation 410, a fusion apparatus selects aclassifier having an optimal or, alternatively, best performance amongcandidate classifiers based on a boosting parameter of training data. Asused in the present disclosure with reference to a performance or aperformance index, the tern “best” refers to a highest or most desirableperformance or performance index from among a plurality of performancesor performance indexes (e.g., a plurality of performances or performanceindexes corresponding to a plurality of candidate classifiers,respectively). The boosting parameter may refer to a training weight.For example, the fusion apparatus may select, in a present iteration, aclassifier to successfully classify the training data not successfullyclassified in a previous iteration based on the boosting parameter.

The fusion apparatus selects a classifier having an optimal or,alternatively, best performance index determined based on the boostingparameter and a result of classifying the training data. At least aportion of the candidate classifiers may be selected for a plurality oftimes. Alternatively, the at least a portion of the candidateclassifiers may not be selected.

In operation 420, the fusion apparatus updates the boosting parameterbased on an output of the classifier selected in operation 410. Theboosting parameter may be determined in a previous iteration. Forexample, the fusion apparatus may update the boosting parameter bydecreasing a value of the boosting parameter of the training data ofwhich a recognition result of the selected classifier is true and/orincreasing the value of the boosting parameter of the training data ofwhich the recognition result of the selected classifier is false. Inoperation 430, the fusion apparatus determines a fusion parameter of theselected classifier based on the boosting parameter updated in operation420.

In an example, the fusion parameters of the candidate classifiers may bedetermined by allowing classifiers to be learned based on boosting. Theboosting refers to a method of iterating an operation of creating a newclassification rule by concentrating on objects improperly classified.The boosting may create a strong prediction model by combining weakprediction models.

The boosting may sequentially generate predictor variables. The boostingmay firstly start at an identical weight with respect to row trainingdata. However, based on an iteration of trainings, the boosting mayallow improperly classified objects to be properly classified, byassigning, in each of the trainings, a high weight to the training data,for example, training data of which a recognition result is false, thatare falsely classified by a classifier, and assigning a low weight tothe training data, for example, training data of which the recognitionresult is true, that are properly classified by the classifier.

For example, 10 items of first training data t_face 1, t_face 2, . . . ,t_face 10 including a face image and 10 items of second training datat_fingerprint 1, t_fingerprint 2, . . . , t_fingerprint 10 including afingerprint image are assumed to be present.

The fusion apparatus may allow the fusion parameters for fusing thecandidate classifiers based on the training data.

In more detail, the fusion apparatus may iteratively select any one ofthe candidate classifiers based on 10 items of the first training dataand 10 items of the second training data. The fusion apparatus mayselect a classifier having an optimal or, alternatively, bestperformance among the candidate classifiers based on the boostingparameter of the training data.

The fusion apparatus may classify first training data using a classifier1(c1), a classifier 2(c2), and a classifier 3(c3) and then obtain aperformance index based on the boosting parameter to correspond to eachof the classifiers 1(c1), 2(c2), and 3(c3). Also, the fusion apparatusmay classify second training data using a classifier 4(c4), a classifier5(c5), and a classifier 6(c6) and then obtain a performance index basedon the boosting parameter to correspond to each of the classifiers4(c4), 5(c5), and 6(c6).

In this example, the performance index based on the boosting parametermay be determined based on whether a result of classifying predeterminedor, alternatively, desired training data is true or false and a boostingparameter of the predetermined or, alternatively, desired training data.In an example, the greater a value of the boosting parameter of thetraining data of which the result of classifying is true, the greater avalue of the performance index based on the boosting parameter.

Hereinafter, the method of determining fusion parameters usingclassifiers, for example, the classifier 1(c1), the classifier 2(c2),and the classifier 3(c3), for face recognition is described as anexample. However, classifiers, for example, the classifier 4(c4), theclassifier 5(c5), and the classifier 6(c6), for fingerprint recognition,may be also used to determine the fusion parameters based on theaforementioned method.

The fusion apparatus may select a classifier having a highestperformance index among the classifiers, for example, the classifier1(c1), the classifier 2(c2), and the classifier 3(c3), for facerecognition. For example, the classifier 2(c2) is assumed to have thehighest performance index. In this example, the selected classifier2(c2) is assumed to recognize that seven, items of training data t_face1, t_face 3, t_face 4, t_face 5, t_face 7, t_face 9, t_face 10 are true,and three items of training data t_face 2, t_face 6, t_face 8 are false,among 10 items of the first training data t_face 1, t_face 2, . . . ,t_face 10 including the face image.

In an initial iteration, the first training data and the second trainingdata may be set to have an identical boosting parameter. Subsequent toselecting a classifier in each iteration, the fusion parameter maymaintain or decrease a value of the boosting parameter of the trainingdata of which the recognition result of the classifier 2(c2) is true.For example, when an initial value of the boosting parameter of thetraining data t_face 1, t_face 3, t_face 4, t_face 5, t_face 7, t_face9, and t_face 10 of which the recognition result is true is “0.2”, thefusion apparatus may decrease the boosting parameter of the trainingdata t_face 1, t_face 3, t_face 4, t_face 5, t_face 7, t_face 9, andt_face 10 to be “0.1.” In an example, the fusion parameter may maintaina value of the boosting parameter of the draining data t_face 1, t_face3, t_face 4, t_face 5, t_face 7, t_face 9, and t_face 10, as “0.2”.

The fusion apparatus may increase the value of the boosting parameter ofthe training data t_face 2, t_face 6, and t_face 8 of which therecognition result of the classifier 2(c2) is false. When an initialvalue of the boosting parameter of the training data t_face 2, t_face 6,and t_face 8 of which the recognition result of the classifier 2(c2) isfalse is “0.2”, the fusion apparatus may increase the value of theboosting parameter of the training data t_face 2, t_face 6, and t_face 8by “0.3” to be updated to be “0.5.” Accordingly, in a subsequentiteration, the fusion apparatus may select a classifier to more properlyclassify the training data of which the recognition result by aclassifier selected in a present iteration is false. The fusionapparatus may determine a fusion parameter of the classifier 2(c2) basedon the boosting parameter updated to be “0.1” or “0.5” in the presentiteration.

Based on the boosting parameter updated in the present iteration, thefusion apparatus may select any one of the classifiers, for example, theclassifier 1(c1), the classifier 2(c2), and the classifier 3(c3), forthe face recognition in the subsequent iteration, and iterate thedetermining of the fusion parameter of the selected classifier. Forexample, the fusion apparatus may select a classifier, for example, theclassifier 1(c1), having an optimal or, alternatively, best performanceamong the candidate classifiers based on a value, for example, “0.1” or“0.5”, of the boosting parameter of the previously updated trainingdata. The fusion apparatus may re-update the boosting parameter updatedto be “0.1” or “0.5” based on an output of the selected classifier, forexample, the classifier 1(c1). Therefore, a boosting parameter updatedin an n-1th iteration may be used in an nth iteration.

As described above, the fusion apparatus may iteratively perform thedetermining of the fusion parameter a predetermined or, alternatively,desired number of times and iteratively perform the aforementionedprocess until a recognition rate determined in the determining of thefusion parameter is to be greater than or equal to a predetermined or,alternatively, desired threshold.

FIGS. 5A and 5B are graphs illustrating examples of distributions ofoutput scores of classifiers based on a first modality and a secondmodality according to at least one example embodiment.

FIG. 5A illustrates a distribution of an output score of a classifierfor face recognition, and FIG. 5B illustrates a distribution of anoutput score of a classifier for fingerprint recognition. In graphs ofFIGS. 5A and 5B, an X-axis indicates an index of a sample, for example,training data, and a Y-axis indicates a Euclidean distance between thesample and an enrolled image. The Euclidean distance indicates that asimilarity is high when the Euclidean distance is closer to be “0”.

In the graphs of FIGS. 5A and 5B, a number of the positive samples forthe face recognition is “60”, and a number of negative samples is also“60”. However, the number 60 is only an example and other numbers ofpositive and/or negative samples can be used. In this example, thepositive samples refer to samples corresponding to an enrolled user, andthe negative samples refer to samples not corresponding to the enrolleduser.

In the graph of FIG. 5A, scores with respect to the positive samples aredistributed in a vicinity of a point at which the Euclidean distance is“0.65” and scores with respect to the negative samples are distributedin a vicinity of a point at which the Euclidean distance is “1.2”.However, outliers are widely distributed outside of a correspondingpoint. Therefore, a reference of authenticating a user based on the facerecognition may be ambiguous.

Since score values of the positive samples and the negative samples arenot clearly classified, a great number of overlapped areas are shown inthe distribution. Therefore, a recognition result may be greatly changedbased on a threshold. In addition, when an FAR increases, a verificationrate (VR) may increase.

In the graph of FIG. 5B, the scores with respect to the positive samplesare distributed at a vicinity of a point at which the Euclidean distanceis “0.02”, and the scores with respect to the negative samples aredistributed in a vicinity of a point at which the Euclidean distance isapproximately “0.11”. In a case of fingerprint recognition, since thescore values of the positive samples and the negative samples areclearly classified, the distribution may be also clearly represented.

The aforementioned features indicate that the positive samples areclearly recognized to be true, however, the positive samples are falselyrecognized to be completely false when a portion of informationassociated with the positive samples is deformed or disappears. Since amethod of correcting samples classified to be true or false by aclassifier does not exist, the VR may not be greatly improved even at ahigh FAR.

When the scores are simply summed in lieu of considering a feature ofeach modality, an accurate recognition result may not be obtained. In anexample, a multi-authentication may be performed by multi-modal fusionbased on a feature of each heterogeneous modality.

In an example, it is possible to perform multi-modal fusion by aclassification based on a gentle-boost scheme, simplify expressions byapproximating a boost result other than using a general gentle-boostscheme, and perform weighted summing of a weight and a score of aclassifier selected through learning. The fusion apparatus may enhanceperformance in speed and volume of memory in lieu of requiring anadditional learning model.

In an example, it is possible to perform the multi-modal fusion by thefusion apparatus including a plurality of classifiers for facerecognition and a plurality of classifiers for fingerprint recognition.The fusion apparatus may select an optimal or, alternatively, bestclassifier for the multi-authentication among the plurality ofclassifiers and obtain an optimal or, alternatively, desirable weight ofthe optical classifiers based on the gentle-boost scheme.

FIGS. 6A through 6C are graphs illustrating examples of a histogramdistribution of a homogeneous modality according to at least one exampleembodiment.

FIGS. 6A through 6C illustrates examples of a histogram distributionquantizing scores of positive samples in each face recognition, forexample, a first modality, fingerprint recognition, for example, asecond modality, and multi-modal fusion, for example, fusion of the facerecognition and the fingerprint recognition. An X-axis indicates anindex of a bin and a Y-axis indicates a number of the positive samplesincluded in each bin.

FIG. 6A is a histogram corresponding to a score of positive samples withrespect to face recognition, FIG. 6B is a histogram corresponding to ascore of positive samples with respect to fingerprint recognition, andFIG. 6C is a histogram corresponding to a score of positive samples inmulti-modal fusion.

Referring to FIG. 6A, in a case of the face recognition, a histogramvariance corresponding to the score of the positive samples has a widthsuch that a false reject rate (FRR) increases based on a change in athreshold. However, in a case of the face recognition, the positivesamples outside of a standard distribution merely exist.

Referring to FIG. 6B, in a case of fingerprint recognition, thehistogram variance corresponding to the score of the positive samples isnarrow such that the FRR based the change in the threshold is relativelyrobust when compared to the face recognition. However, in thefingerprint recognition, a large number of the positive samples mayexist outside of the standard distribution, and the FRR may increase bythe positive samples deviated from the standard distribution.

In an example, it is possible to fuse heterogeneous modalities, forexample, face recognition and fingerprint recognition, based on featuresof each modality in FIGS. 6A and 6B. FIG. 6C illustrates the histogramcorresponding to the score of the positive samples with respect to themulti-modal fusion performing fusion on the heterogeneous modalities.

Referring to FIG. 6C, when the heterogeneous modalities are fusioned orfused, the variance of the histogram has a distribution in a middle ofthe distributions in FIGS. 6A and 6B, and the positive samples outsideof the standard distribution merely exist. Therefore, performance of theFRR may greatly enhance.

FIG. 7 is a flowchart illustrating a user authentication method based ona multi-modal fusion method according to at least one exampleembodiment.

Referring to FIG. 7, an authentication apparatus may authenticate a userusing the fusion parameters determined by the fusion apparatus describedwith reference to FIGS. 1 through 6. The authentication apparatus may beprovided in a software module, a hardware module, or variouscombinations thereof. For example, the fusion apparatus may be may beimplemented based on various methods such as a single processor-basedsystem, a special purpose hardware-based system, a system in which aprocessor and an HWA are combined, and a cloud computing system. Thefusion apparatus may include a user terminal, for example, a smartphone.

In operation 710, the authentication apparatus receives a first inputimage including information on a first modality. In operation 720, theauthentication apparatus receives a second input image includinginformation on a second modality. For example, the first modality may bea face of a user and the second modality may be a fingerprint of theuser. In this example, the first input image may be a face image and thesecond input image may be a fingerprint image. The fingerprint image maybe a portion image including information on a portion of an entirefingerprint of the user.

In operation 730, the authentication apparatus determines at least onefirst score by processing the first input image based on at least onefirst classifier based on the first modality. In operation 740, theauthentication apparatus determines at least one second score byprocessing the second input image based on at least one secondclassifier based on the second modality.

The authentication apparatus may divide the first input image and thesecond input image into a plurality of sub-blocks. The plurality ofsub-blocks may correspond to an area or a portion of the first inputimage and the second input image.

For example, when the first input image is the face image, theauthentication apparatus may extract a plurality of subareas from theface image. The plurality of subareas may be determined in advance as acontour of a face, an eye, and a nose. When the second input image isthe fingerprint image, the authentication apparatus may divide thefingerprint image into a plurality of blocks. The plurality of blocksmay have predetermined or, alternatively, desired sizes andpredetermined or, alternatively, desired shapes.

The authentication apparatus may determine a score of the sub-blocksdivided using a corresponding classifier. For example, when the firstinput image is the face image, the at least one first classifier may bedetermined to correspond to a plurality of predetermined or,alternatively, desired subareas among the face image. In this example,the at least one first classifier includes classifiers corresponding tothe plurality of subareas, for example, a contour of a face, an eye, anda nose.

When the second input image is the fingerprint image, at least onesecond classifier may correspond to at least a portion of the pluralityof blocks. For example, the at least one second classifier may bedetermined based on a result of arranging matching scores betweenenrolled images and blocks to divide the fingerprint image. The at leastone second classifier may correspond to a predetermined or,alternatively, desired number of pairs of the enrolled images and theblocks having high matching scores between the enrolled images and theblocks. The authentication apparatus may calculate the matching scoresbetween a plurality of enrolled images and a plurality of blocks andselect a predetermined or, alternatively, desired number of best scores.Each of the plurality of enrolled images may be portion images includinginformation on a portion of a fingerprint of a pre-enrolled user. Inthis example, the at least one second classifier may refer toclassifiers determining the selected best scores.

In an example, the at least second classifier may be provided in asingle module. In this example, the single module may sequentiallydetermine the matching scores between the plurality of enrolled imagesand the plurality of blocks. For example, the single module maysequentially determine matching scores between the plurality of enrolledimages and a first block, and sequentially determine matching scoresbetween the plurality of enrolled images and a second block. The singlemodule may determine the matching scores between the plurality ofenrolled images and each of the blocks based on the aforementionedmethod.

In operation 750, the authentication apparatus authenticates a userbased on the at least one first score, the at least one second score, afirst fusion parameter of the at least one first classifier, and asecond fusion parameter of the at least one second classifier.

The first fusion parameter and the second fusion parameter maycorrespond to values determined based on features of each modality. Forexample, when the first input image is a face image, a position of asubarea in the face corresponding to the first input image issignificant. Accordingly, the first fusion parameter may reflect a firstpriority based on the position of subarea in the face. In addition, whenthe second input image is a portion fingerprint image, a position of theportion fingerprint image in an entire fingerprint or a position of ablock in the portion fingerprint image are not significant. Accordingly,the second fusion parameter may less reflect, compared to the firstpriority, or not reflect a second priority based on the position of theportion fingerprint image in the entire fingerprint or the position ofthe block in the portion fingerprint image.

In this example, the first fusion parameter and the second fusionparameter may be determined based on a boosting scheme in a learningstage. For example, the first fusion parameter and the second fusionparameter may be determined by selecting any one of at least one firstclassifier and at least one second classifier in each iteration of theboosting scheme.

For example, at least one of the first fusion parameter and the secondfusion parameter may be determined based on the following process. Theauthentication apparatus selects a classifier having an optimal or,alternatively, best performance among candidate classifiers based on aboosting parameter of training data and updates the boosting parameterbased on an output of the selected classifier. The authenticationapparatus may determine at least one of the first fusion parameter andthe second fusion parameter by determining a fusion parameter of theselected classifier based on the updated boosting parameter.

Repeated descriptions will be omitted for increased clarity andconciseness because the descriptions provided with reference to FIGS. 1through 6 are also applicable to FIG. 7.

The units and/or modules described herein may be implemented usinghardware components and software components. For example, the hardwarecomponents may include microphones, amplifiers, band-pass filters, audioto digital convertors, and processing devices. A processing device maybe implemented using one or more hardware device configured to carry outand/or execute program code by performing arithmetical, logical, andinput/output operations. The processing device(s) may include aprocessor, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a field programmable array, a programmablelogic unit, a microprocessor or any other device capable of respondingto and executing instructions in a defined manner. The processing devicemay run an operating system (OS) and one or more software applicationsthat run on the OS. The processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For purpose of simplicity, the description of a processingdevice is used as singular; however, one skilled in the art willappreciated that a processing device may include multiple processingelements and multiple types of processing elements. For example, aprocessing device may include multiple processors or a processor and acontroller. In addition, different processing configurations arepossible, such a parallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct and/or configure the processing device to operateas desired, thereby transforming the processing device into a specialpurpose processor. Software and data may be embodied permanently ortemporarily in any type of machine, component, physical or virtualequipment, computer storage medium or device, or in a propagated signalwave capable of providing instructions or data to or being interpretedby the processing device. The software also may be distributed overnetwork coupled computer systems so that the software is stored andexecuted in a distributed fashion. The software and data may be storedby one or more non-transitory computer readable recording mediums.

The methods according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations of the above-describedexample embodiments. The media may also include, alone or in combinationwith the program instructions, data files, data structures, and thelike. The program instructions recorded on the media may be thosespecially designed and constructed for the purposes of exampleembodiments. Examples of non-transitory computer-readable media includemagnetic media such as hard disks, floppy disks, and magnetic tape;optical media such as CD-ROM discs, DVDs, and/or Blue-ray discs;magneto-optical media such as optical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas read-only memory (ROM), random access memory (RAM), flash memory(e.g., USB flash drives, memory cards, memory sticks, etc.), and thelike. Examples of program instructions include both machine code, suchas produced by a compiler, and files containing higher level code thatmay be executed by the computer using an interpreter. Theabove-described devices may be configured to act as one or more softwaremodules in order to perform the operations of the above-describedexample embodiments, or vice versa.

Example embodiments of the inventive concepts having thus beendescribed, it will be obvious that the same may be varied in many ways.Such variations are not to be regarded as a departure from the intendedspirit and scope of example embodiments of the inventive concepts, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A user authentication method comprising:receiving a first input image including information on a first modality;receiving a second input image including information on a secondmodality; determining at least one first score by processing the firstinput image based on at least one first classifier, the at least onefirst classifier being based on the first modality; dividing the secondinput image into a plurality of blocks which correspond, respectively,to a plurality of second classifiers; determining best scores from amongmatching scores between enrolled images and the plurality of blocks;determining at least one second score by processing the second inputimage based on at least one second classifier from among the pluralityof second classifiers, the plurality of second classifiers being basedon the second modality, wherein the at least one second classifier isdetermined based on the determined best scores; and authenticating auser based on the at least one first score, the at least one secondscore, a first fusion parameter of the at least one first classifier,and a second fusion parameter of the at least one second classifier. 2.The method of claim 1, wherein the determining of the at least one firstscore comprises: extracting a subarea from the first input image; anddetermining a score of the subarea using a classifier corresponding tothe extracted subarea among the at least one first classifier.
 3. Themethod of claim 1 wherein, the first modality is a modality of a firsttype and the second modality is a modality of a second type, the firsttype and the second type are each at least one of a face of the user, afingerprint of the user, an iris of the user, or a vein of the user, andthe first type differs from the second type.
 4. The method of claim 1,wherein the first fusion parameter and the second fusion parameter aredetermined in advance based on a boosting scheme.
 5. The method of claim4, further comprising: executing the boosting scheme such that theboosting scheme includes, selecting a classifier having an optimalperformance among candidate classifiers based on a boosting parameter oftraining data, updating the boosting parameter based on an output of theselected classifier, and determining at least one of the first fusionparameter and the second fusion parameter by determining a fusionparameter of the selected classifier based on the updated boostingparameter.
 6. The method of claim 1, wherein the authenticating of theuser comprises: performing weighted summing of the first score and thesecond score based on the first fusion parameter and the second fusionparameter; and authenticating the user by performing a comparisonoperation based on a result of the weighted summing and a firstthreshold.
 7. The method of claim 1, further comprising: determining theat least one first classifier to correspond to at least one subarea inthe first input image.
 8. The user authentication method of claim 1,wherein the best scores are the highest scores from among the matchingscores, and a total number of the best scores is less than a totalnumber of the matching scores.
 9. A non-transitory computer-readablemedium storing program code including instructions that, when executedby one or more processors, cause the one or more processors to performthe method of claim
 1. 10. A multi-modal fusion method comprising:determining fusion parameters of candidate classifiers by iterativelyselecting classifiers from among candidate classifiers over a pluralityof iterations such that the iterative selecting includes, selecting,during a first iteration of the plurality of iterations, a classifierhaving a best performance among the candidate classifiers based on aboosting parameter of training data, updating the boosting parameterbased on an output of the selected classifier, and determining a fusionparameter of the selected classifier based on the updated boostingparameter, the determined fusion parameter being a weight valuecorresponding to the selected classifier, the candidate classifiersincluding, at least one first classifier configured to authenticate auser based on a first modality, and at least one second classifierconfigured to authenticate the user based on a second modality, whereinthe at least one second classifier is based on a result of arrangingmatching scores between enrolled biometric information and blocks todivide input biometric information.
 11. The method of claim 10, whereinthe boosting parameter is determined during a second iteration fromamong the plurality of iterations, the second iteration occurring beforethe first iteration.
 12. The method of claim 10, wherein the selectingof the classifier comprises: selecting a classifier having an bestperformance index determined based on the boosting parameter and aresult of classifying the training data.
 13. The method of claim 10,wherein the updating of the boosting parameter comprises: decreasing avalue of the boosting parameter of data from among the training datathat was properly classified by the selected classifier; and increasingthe value of the boosting parameter of data from among the training datathat was improperly classified by the selected classifier.
 14. Themethod of claim 10, further comprising: determining a final parameterset based on the fusion parameters when the determining of the fusionparameters is iteratively performed a first number or times.
 15. Themethod of claim 10, further comprising: determining a final parameterset based on the fusion parameters when a recognition rate determined inthe determining of the fusion parameters is greater than or equal to afirst threshold.
 16. The method of claim 10 wherein, the first modalityis a modality of a first type and the second modality is a modality of asecond type, the first type and the second type are each at least one ofa face of the user, a fingerprint of the user, an iris of the user, anda vein of the user, and the first type differs from the second type. 17.The method of claim 10, further comprising: determining the at least onefirst classifier to correspond to at least one subarea in an inputimage.